Method and device for evaluating images, operational assistance method and operating device

ABSTRACT

A method for evaluating images and, in particular, for evaluating correspondence hypotheses of images. The method includes (i) providing a hypothesis matrix of correspondence hypotheses between first and second images, each given as a corresponding image matrix, (ii) evaluating the hypothesis matrix and conditional verification of the image correspondence hypotheses and (iii) providing verified image correspondence hypotheses in a correspondence matrix of image correspondences as the evaluation result, the hypothesis matrix being evaluated by forming and evaluating a histogram with respect to the values of the component for each element of the hypothesis matrix for at least one component of correspondence hypotheses.

FIELD

The present invention relates to a method and a device for evaluatingimages and, in particular, correspondence hypotheses of images, anoperational assistance method and, in particular, a driver assistancemethod, as well as an operating device and, in particular, a vehicle.The present invention also relates to a computer program and to amachine-readable memory medium.

BACKGROUND INFORMATION

Image processing is used in many technical fields for controllingdevices and processes, for example, also in the automotive industry inso-called driver assistance systems. In this context, images arecaptured, for example, and used as a basis for a control process. Forthis purpose, the images must be evaluated. When evaluating images,so-called correspondences are frequently generated and used as a basisfor the evaluation. Such correspondences describe assignments betweencoordinates in a first image and coordinates in a second imagepixel-by-pixel and in a temporal and/or spatial direction. A temporalrelationship may be images recorded in succession, a spatialrelationship may be spatially separated images, which may also berecorded simultaneously, for example, in stereo vision. When derivingcorrespondences, correspondence hypotheses are initially formed. Thecorrespondence hypotheses must be verified or falsified. This process ismemory-intensive, computationally-intensive and/or time-intensive, inparticular, in the case of high-resolution images.

SUMMARY

A example method for evaluating images according to the presentinvention may have the advantage over the related art that withcomparatively little effort correspondence hypotheses for a pair ofimage may be verified, i.e., confirmed or rejected with a high degree ofreliability. This may be achieved according to the present invention. Inaccordance with and example embodiment of the present invention a methodfor evaluating images and, in particular, for evaluating correspondencehypotheses of images is provided, which includes the steps:

-   (i) providing correspondence hypotheses between first and second    images each given as a corresponding image matrix as elements in a    corresponding hypothesis matrix,-   (ii) evaluating the hypothesis matrix and conditionally verifying    the correspondence hypotheses and-   (iii) providing verified image correspondence hypotheses as image    correspondences in a correspondence matrix as an evaluation result.

The hypothesis matrix is evaluated according to the present invention byforming and evaluating a histogram for each element as a referenceelement of the hypothesis matrix in the surroundings of the respectiveelement for at least one component of the correspondence hypotheses. Theresult of these measures according to the present invention is thathypotheses for elements or pixels of the underlying images may beevaluated via their surroundings in the image or in the matrix of thecorrespondence hypotheses and consequently with a high degree ofreliability and, thus verified or falsified.

In this case, the present invention exploits the fact that a correctcorrespondence in the hypothesis image is confirmed by correspondencesin the vicinity, because they are similar in value, i.e., for example,with respect to direction or length, whereas false correspondences donot exhibit this similarity. With the approach according to the presentinvention, a reliable and efficient evaluation and separation of thecorrect and of the false correspondences is effectuated.

In this context, “conditional verification” in the broader sense meansthat when checking one or multiple predefined condition(s), it isdecided according to the present invention whether a respectivecorrespondence hypothesis is confirmed and maintained, i.e., verified inthe narrower sense, or else denied or rejected, i.e., is falsified.

The present invention is applicable in connection with any form ofcameras, monitoring cameras and/or metrology cameras, for example,—butnot only—on or in vehicles in general, in particular, in connection withmotor vehicles. Also included in this case are applications of thepresent invention in connection with television cameras, consumercameras, drone cameras, human-machine interaction cameras, etc.

Preferred refinements of the present invention are described herein.

The totality of the correspondence hypotheses are also understood withinthe context of the present invention as a hypothesis image or hypothesismatrix, the individual components thereof are also referred to aselements or pixels. The individual correspondence hypotheses may includeone or also multiple scalar component(s), for example, values forvarious movement directions u, v in the hypothesis image and/or in aprojection plane in the underlying space. Thus, they may bevector-valued. Furthermore, the correspondence hypotheses may also bebased on the concept of the optical flow, also referred to as OF, and/orreflect a stereo disparity, but this is not mandatory.

The structure of one or of multiple histograms with respect to thehypothesis matrix may be implemented in various ways.

In one preferred specific embodiment of the method according to thepresent invention, a histogram for a given respective element of thehypothesis matrix is formed by classifying, and summing or summing in aweighted manner the respective components of the correspondencehypotheses in a given histogram classification for—in particular,all—elements of the hypothesis matrix in the interior of thesurroundings of the given element.

In this case, it is not mandatory that all elements of the hypothesismatrix encompassed by the respective surroundings of each given elementof the hypothesis matrix are evaluated or even considered, i.e., forexample, read out, for forming the histogram. Instead, it is possible asa function of the respective application and situation to make asuitable choice, for example, in order to accelerate the methodaccording to the present invention or in order to reduce the computingtime. This is further explained in detail below.

The surroundings for a given respective element of the hypothesis matrixmay be defined by different means.

The method according to the present invention is particularly flexiblystructured if according to one preferred refinement, when forming arespective histogram, the surroundings for a given respective element ofthe hypothesis matrix are provided in each case with a window wholly orpartially overlaying the hypothesis matrix, in particular, in the mannerof a rectangle, a polygon, an oval, an ellipsis or a circle.

The given, respective element of the hypothesis matrix described in thiscontext virtually forms a reference element or reference pixel. Thisreference element or reference pixel is assigned the respectivelygenerated histogram and is evaluated for this histogram.

The method according to the present invention operates particularlyreliably when all elements of the hypothesis matrix are captured duringthe evaluation and, in particular, during the formation of histograms.

In this case, it is advantageous if the respective surroundings or arespective window is used, which is identical or similar for allelements of the hypothesis matrix with respect to form and/or extent,because a high degree of comparability of the results for variouselements of the hypothesis matrix occurs under these circumstances.

In order to save time-critical or computationally-intensive or memoryaccess-intensive working steps for the purpose of sequence optimizationwith respect to the operational sequence of the method according to thepresent invention, it is provided in another advantageous refinementthat respective surroundings or a respective window are designed assliding surroundings or as a sliding window. In the application, thismeans that the sliding window is placed as an ideal construct above thehypothesis matrix where it is shifted, in order to establish therespective reference element and the surroundings that are to be usedfor calculating the histogram. The sliding movement of the window thentakes place stepwise, for example, stepwise in the row direction columnby column or in the column direction row by row, an existing histogramthen being updated for a new reference element, namely by adding orentering newly arriving elements of the hypothesis matrix and omittingor removing discontinued elements of the hypothesis matrix.

When evaluating a histogram for a given respective element as areference element of the hypothesis matrix, an evaluation value may begenerated based on the histogram. This value is then used as the actualevaluation of the correspondence hypothesis belonging to the referenceelement or reference pixel.

The evaluation value may be generated, for example, by reading out oneor multiple value(s) of the histogram in at least one histogram intervalfor the correspondence hypothesis assigned to the given element or forat least one component thereof and by summing them or summing them in aweighted manner to form the evaluation value.

In the process, one or multiple values from one or from multipleadjacent histogram interval(s) assigned to the histogram interval orfrom the surroundings of adjacent histogram intervals may preferablyalso be taken into consideration. In this way, it is possible to easilytake the hypothesis surroundings for the correspondence hypothesis ofthe reference pixel into consideration, in order to verify or to falsifythe correspondence hypothesis of the reference pixel.

In another advantageous refinement of the method according to thepresent invention, it is detected when verifying a verification of acorrespondence hypothesis for a respective given element as a referenceelement of the hypothesis matrix or for a component thereof, when theevaluation value at least reaches a local or global predefined thresholdvalue. This means specifically that it is checked whether the evaluationvalue for the correspondence hypothesis of the reference pixel is belowor above a threshold value or at least reaches this value. In the firstcase, the correspondence hypothesis is rejected as an outlier, in thelatter case, it is verified, confirmed and then adopted directly or inmodified form as correspondence into the correspondence matrix or intothe correspondence image.

Alternatively or in addition, it may be provided that when verifying acorrespondence hypothesis for a respectively given element as areference element or reference pixel of the hypothesis matrix or for acomponent thereof, an assigned and verified correspondence or acomponent thereof is established in each case by an evaluation value andas such is preferably entered as a corresponding element into thecorrespondence matrix or into the correspondence image.

In specific embodiments, various technical measures may be taken inorder to accelerate the method according to the present invention duringthe execution and/or to reduce the processing effort required in eachcase.

This may also be understood to mean, in particular, that the number ofoperations is reduced, for example, the number of memory accesses, ofadditions and/or of multiplications. This may be utilized for a temporalacceleration in terms of a more rapid implementation for reducing theresources provided for the implementation or for both simultaneously.

It is therefore particularly advantageous if, according to oneembodiment of the method according to the present invention foraccelerating the evaluation of the hypothesis matrix and/or of theconditional verification of the correspondence hypotheses, arespectively instantaneously generated histogram, which is stored, inparticular, in a histogram memory, is used for the parallel and/orserial verification of multiple correspondence hypotheses, inparticular, for evaluating correspondence hypotheses of a plurality ofelements of the hypothesis matrix as reference elements or as referencepixels of the hypothesis matrix which, in particular, may be directlyadjacent to one another.

In this context, it is also advantageous if according to anotherembodiment of the method according to the present invention, a histogrammemory used is organized in such a way that an underlying clock cycle ofthe processing is sufficient in order to read out the value for arespectively updated element as a reference element of the hypothesismatrix from the histogram, as well as values for one or for multipleadditional interval(s) of the histogram.

This may advantageously take place by (i) calculating an underlying readaddress by adding or subtracting the value 1 with respect to the valuefor the updated element stored in the hypothesis matrix and/or (ii) byclustering intervals of the histogram, in particular, with respect to adual port of the underlying histogram memory.

Alternatively or in addition, it is possible for accelerating theevaluation of the hypothesis matrix and/or the conditional verificationof the correspondence hypotheses, to carry out the conditionalverification in parallel, in particular, simultaneously, and/or seriallyfor a plurality of correspondence hypotheses for a correspondingplurality of elements in terms of reference elements of the hypothesismatrix.

A given filling state of a histogram may be utilized in order to verifymultiple hypotheses—namely for the reference position in the hypothesisimage and for close neighbors. Since the updating of the histogram iscomplex and the number of the required updates may be reduced with thisprocess—for example, to ¼—the result is a savings in the processingeffort, for example, in terms of the number of memory accesses,additions, weightings and the like.

It is also possible that not all elements of the hypothesis matrix inthe interior of the surroundings of the respectively given element aretaken into consideration as a reference element of the hypothesis matrixand, in particular, of the underlying sliding window when updating ahistogram.

In this case, for example, it is possible to use fewer rows than isequivalent to the maximum row number of the sliding window, inparticular, using a row number of 4.

The present invention further relates to an operational assistancemethod and, in particular, to a driver assistance method for a deviceand, in particular, for a vehicle, in which images are captured and areevaluated using a method according to a method according to the presentinvention, and in which a result of the evaluation is used incontrolling the operation of the device.

According to another aspect of the present invention, an example devicefor evaluating images and, in particular, for evaluating correspondencehypotheses of images is also specified, which is configured to carry outan operational assistance method or driver assistance method accordingto the present invention or a method according to the present inventionfor evaluating images and, in particular, correspondence hypotheses forimage pairs.

The device according to the present invention may be designed, inparticular, as an ASIC, as a freely programmable digital signalprocessing device or as a combination thereof.

In addition, a computer program is also specified by the presentinvention, which is configured to carry out a method according to thepresent invention when it is executed on a computer or on a digitalsignal processing unit.

Furthermore, the present invention also provides a machine-readablememory medium on which the computer program according to the presentinvention is stored.

BRIEF DESCRIPTION OF THE DRAWINGS

Specific embodiments of the present invention are described in detailwith reference to the figures.

FIG. 1 schematically shows one specific embodiment of the methodaccording to the present invention in the form of a flow chart andillustrates the correspondence between the underlying images, thehypothesis matrix and the correspondence matrix.

FIGS. 2-5 schematically illustrate possible connections between anunderlying hypothesis matrix and an applied sliding window.

FIG. 6 shows graphs, each of which represents a histogram for onecomponent of an underlying correspondence matrix.

FIG. 7 also schematically illustrates a possible connection between anunderlying hypothesis matrix and an applied sliding window.

FIGS. 8-10 illustrate measures that may be taken to accelerate specificembodiments of the method according to the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Exemplary embodiments of the present invention and of the technicalbackground are described in detail below with reference to FIGS. 1through 10. Identical and equivalent elements and components, as well aselements and components acting identically or equivalently, areidentified with the same reference numeral. The detailed description ofthe identified elements and components is not rendered each time theyoccur.

The features and additional properties depicted may be isolated from oneanother in arbitrary form and may be arbitrarily combined with oneanother, without departing from the core of the present invention.

FIG. 1 schematically shows one specific embodiment of the method Saccording to the present invention for evaluating images B1 and B2 inthe form of a flow chart and illustrates the correspondence betweenunderlying images B1, B2 of hypothesis matrix 10 and of correspondencematrix 100.

In a first step S1 of method S according to the present invention forevaluating images B1 and B2, a hypothesis matrix 10 is initiallyprovided, which may also be understood to be a hypothesis image andincludes as elements correspondence hypotheses [u, v]—here withcomponents u and v—, which are to be evaluated in order to decidewhether a respective correspondence hypothesis actually results in acorrespondence or must be rejected as an outlier.

In this case, [u, v] may be understood to be a vector, whose componentsu and v each have the unit pixel. Such a vector [u, v] may be added toor subtracted from a vector [x, y]. Vector [x, y] in this case mayrepresent a position (as a pixel coordinate) in image B1 or B2. The sum[x, y]+[u, v] or the difference [x, y]−[u, v] may represent a positionin the respectively other image. Thus, a vector [u, v] at a position [x,y] represents a relationship in terms of an assignment to a position inthe respectively other image. This relationship may correspond to acorrespondence: this is the case if the same scene point is representedat both image positions. The point is to find such correctcorrespondences. In the event of confirmation, [u, v] is stored at theposition of [x, y] in correspondence matrix 100 and is identified asvalid. In the event of the rejection, this is noted accordingly at theposition of [x, y]. To identify as valid/invalid, one bit per position[x, y] may be provided or a specific vector [u, v], which is situated atthe edge of the value range of u and v.

A vector [u, v] generally does not include integer values. Vector [x, y]on the other hand preferably includes integer values.

To decide whether or not a correspondence hypothesis is to be evaluatedas an outlier, hypothesis matrix 10 or the hypothesis image areevaluated in a second step S2 and verified in a subsequent step S3 onthe basis of an evaluation result. This means, in particular, that foreach correspondence hypothesis [u, v] or for each of the components uand v, it is decided whether the correspondence hypothesis [u, v] orcomponent u, v is rejected and denied or is confirmed.

The totality of all confirmed correspondence hypotheses [u, v] thenforms corresponding correspondence matrix 100, which may also bereferred to as a correspondence image and which contains the confirmedcorrespondence as elements or pixels.

According to the present invention, evaluation S2 of hypothesis matrix10 takes place by forming in a step S2-1, for each element 1 ofhypothesis matrix 10, functioning here as a so-called reference elementor reference pixel, one or multiple histogram(s) 63, 64 for a respectiveassigned correspondence hypothesis [u, v] or for components u, v whichform the correspondence hypothesis with respect to the values ofcomponents u, v of correspondence hypothesis [u, v].

These histograms 63, 64 are then evaluated in a subsequent step S2-2.

FIG. 1 also shows that images B1, B2, hypothesis matrix 10, alsounderstood to be a hypothesis image, and correspondence matrix 100, alsounderstood to be a correspondence image, may each be represented asmatrices corresponding to one another and/or as correspondinglyconfigured memories or memory areas. This serves as a simplifiedrepresentation, but is not mandatory as long as the respective 1-to-1correspondence between at least one of images B1, B2, of hypothesismatrix 10 and of correspondence matrix 100 and their respectivecomponents or pixels may be established.

FIGS. 2 through 5 schematically illustrate possible connections betweenan underlying hypothesis matrix 10 and an applied sliding window 20,which defines or forms the surroundings for eliciting a histogram 63, 64for a respectively given reference pixel 1.

In all specific embodiments depicted in FIGS. 2 through 5, the totalityof correspondence hypotheses [u, v] is represented as rectangularhypothesis matrix 10 including a particular number of columns and aparticular number of rows. Hypothesis matrix 10 may also be understoodto be a hypothesis image and includes elements 11, which may also bereferred to as pixels. Thus, each element 11 of hypothesis matrix 10includes a correspondence hypothesis [u, v], which may be made up of onecomponent or, as here, of a plurality of components u, v.

In this case, however, not every correspondence hypothesis [u, v] musthave a valid content, because it is possible, for example, that nohypothesis was able to be determined for a particular image position,for example, because the respective scene point is visible in image B1,but is concealed in image B2 or is undiscoverable for other reasons.

It is therefore provided that a piece of valid/invalid information maybe specified for each memory location of hypothesis matrix 10. Toidentify as valid/invalid, one bit per position [x, y] may be providedor a specific vector [u, v], which is situated, for example, at the edgeof the value range of u and v.

Hypotheses identified as invalid are subsequently consistently ignored.Hence, they have no influence on histograms 63, 64. A hypothesisidentified as invalid also cannot lead to a result in correspondencematrix 100.

The type of correspondence hypotheses [u, v] and their components u, v,is a function of the underlying method S1 of providing correspondencehypotheses [u, v]. This includes the physical nature of the underlyingcaptured images B1 and B2 and the nature of the mechanisms themselvesfor forming correspondences. Thus, the correspondence formation may bebased on a concept of the optical flow (OF) and/or on aspects of thestereo disparity and optionally other aspects.

In the examples shown in FIGS. 2 through 5, the respective surroundingsof a reference pixel 1 of underlying hypothesis matrix 10 is provided bya sliding window 20 having a width B and a height H, includingcorresponding numbers of columns and rows. Sliding window 20 covers ineach case an area 15 of hypothesis image 10 and includes correspondingelements 21, which may also be referred to as pixels. An element in thiscase, which is identified with 1 or 26, is marked and serves as areference pixel or as a reference element 1, 26 in relation to slidingwindow 20. Reference element or reference pixel 1, 26 may be a centralpixel in relation to the window.

Histograms 63, 64 described below are always ascertained on the basis ofsliding window 20 and of area 15 of elements 11, 21 of underlyinghypothesis matrix 10 covered by sliding window 20.

Histograms 63, 64 to be described below are advantageously updated by asliding displacement, for example, along the row direction, i.e.horizontally in the direction of arrow 23 in FIGS. 2 through 5. Duringthis sliding movement, i.e., during the displacement, the pixels orelements 24 of hypothesis matrix 10 drop out of sliding window 20 on theleft side or rear side 17, whereas pixels 22 reach under or into slidingwindow 20 on the right side, which is also referred to as front side 16.

A histogram 63, 64 is updated by removing departing pixels 24 andclosing out their counters or weights from histogram 63, 64 and byregistering the counters or weights of incoming pixels 22 into histogram63, 64.

Whereas sliding window 20 has a rectangular shape 18-1 in FIG. 2, theshape 18-2 in the specific embodiment according to FIG. 3 is that of anoval.

FIGS. 4 and 5 illustrate the handling of reference pixels 1, 26 and, inparticular, of sliding windows 20 when reference pixel 1, 26 is locatedat the edge of hypothesis image 10, so that if reference pixel 1, 26functions in this case as a central pixel of sliding window 20, slidingwindow 20 projects beyond the edge of hypothesis image 10.

FIG. 5 in this case also illustrates a back and forth movement ofsliding window 20 in directions 23 and 23′ with an interim row change invertical direction 27.

These connections are further explained in the following paragraphs.

FIG. 6 shows graphs 60, each of which represents a histogram 63, 64 foran element as a reference pixel or reference element 1, 26 of anunderlying correspondence matrix 10, specifically for components u and vof correspondence hypothesis [u, v].

FIG. 7 also schematically illustrates a possible connection between anunderlying correspondence matrix 10 and an applied sliding window 20.

FIGS. 8 through 10 schematically illustrate measures that may be takenfor accelerating specific embodiments of method S according to thepresent invention. These measures are explained in detail in thefollowing paragraphs and relate, in particular, to the simultaneousevaluation of correspondence hypotheses [u, v] for a plurality ofadjacent reference pixels 1, 26 of underlying hypothesis matrix 10.

These and other features and properties of the present invention arefurther explained with reference to the following statements.

The topic of correspondence formation is encountered in the area ofcomputer vision, i.e., in machine-aided or computer-aided seeing, inparticular, in the case of optical flow and stereo disparity.

In the cases of optical flow, also referred to above and hereinafter asOF, correspondences are formed in a temporal direction by determiningassignments between coordinates in a first image and coordinates in asecond image. Such a correspondence then indicates how the projection ofa point in the 3D scene moves further temporally into a 2D image from anold coordinate to a new coordinate.

In the process, the movement in the image may be caused by the movementof the scene point, by the movement of the camera or by bothsimultaneously.

In the case of stereo vision, the two images are captured approximatelysimultaneously by two cameras situated at different locations. Therelative arrangement of the cameras in this case is generally fixed andknown. The correspondence formation allows the distance to the point inthe 3D scene to be determined with the aid of triangulation.

A wide variety of methods for correspondence formation are known, theseare presupposed here.

The results of the correspondence formation serve initially ashypotheses, thus, are potential correspondences, some or many of whichmay be erroneous and are then identified as outliers.

In this case, one or multiple hypotheses or also no hypothesis per imagecoordinate may be present.

One object of the present invention is to provide a preferably reliablemethod for verifying correspondence hypotheses, so that preferably manyoutliers are identified and, if necessary, eliminated and the correcthypotheses—as valid values or inliers—are largely maintained.

If multiple different hypotheses per image coordinate are present, thengenerally at most one of them is correct.

A core aspect of the present invention is the creation of ahistogram-based method of verification of correspondence hypotheses. Theunderlying assumption in this case is that local vicinities of imagepoints are usually maintained in the transition from a first image to asecond image.

A vector—for example, a vector of the optical flow, which is alsoreferred to as an optical flow vector,—which combines the coordinates ofa scene point projected into both images with one another, thereforebehaves usually similarly to the vectors at the adjacent imagepositions. “Similar” in this case means that it has a similar length anda similar orientation or has similar vector components, for example,with respect to the horizontal component u and the vertical component v.

In natural images which originate, for example, from a driver assistancecamera, robotics camera or monitoring camera, this assumption issatisfied usually in most of the images. At the edges of objects, forexample, at depth jumps in the scene, in particular, it may be violated,however.

Flow fields ascertained with respect to correspondingly successiveimages may be visualized in a different way, for example, with the aidof a color coding, a respective color indicating the flow direction, forexample, yellow: downward, blue: upward, pink: to the left, green: tothe right. The color saturation indicates the length of the flow vector,so that the zero vector, for example, is depicted as white. Black areasmay then be utilized for those cases for which no value for the opticalflow is able to be determined.

In natural images—i.e., for example, in the optical imaging of naturalscenes—it is the case that adjacent pixels of temporally successiveimages or, in the case of stereo vision, images assigned to one another,have mostly similar flow vectors or correspondence vectors, and thatthus the assumption of the similarity in the local vicinity is properlysatisfied. This assumption is advantageously utilized by the presentinvention.

As was previously mentioned above, exceptions apply, in particular, atobject edges, for example, at the edge between an imaged pedestrian inthe foreground of the image and the equally imaged background of theimage or at the edge of an imaged cyclist in the foreground of the imageand the background and the like. Here, however, the assumption isfrequently always at least partially satisfied, because at least a partof the adjacent pixels exhibits similar correspondences, for example,flow vectors. Such cases are also covered by the present invention.

In conjunction with the present invention, the optical flow usually ismentioned by way of example as an example of the correspondenceformation. However, other applications, for example, stereo systems,multi-camera systems and more are also covered. It is not necessary inthis case for the underlying camera systems to be calibrated. Thus, forexample, the epipolar rectification—based on a calibration—frequentlyused in stereo systems is also not necessary in order to be able toapply the present invention. The present invention may even be appliedto much more general examples of correspondence formation, such aslocating similar text sections in a document or across documentboundaries.

By entering the correspondence hypotheses of the local surroundings and,in particular, their values for the movement in the horizontal directionor u-direction, and in the vertical direction or v-direction, in eachcase in a single and then one-dimensional histogram or in a combined,shared and then two-dimensional histogram, peaks or maxima are formedwith suitable scaling in the histogram, which originate from thedominant local movement. This is depicted in FIG. 6, for example, forthe use of two separate, one-dimensional histograms.

There exist therefore multiple possibilities for selecting suitablehistograms:

-   Two one-dimensional histograms:    -   A one-dimensional histogram is used for the first movement        component, for example, the horizontal movement u. The second        one-dimensional histogram is used for the movement component        perpendicular thereto, for example, the vertical movement        component v.    -   Movement components u, v need not be oriented on or in parallel        to the image axis of the respectively underlying images.        Diagonal components, for example, or any other orientation, are        also possible.    -   The components may also represent other variables, for example,        the direction and the length of a flow vector.-   A multi-dimensional histogram (composite histogram):    -   In a multi-dimensional histogram, which is also referred to as a        composite histogram, for example, in a two-dimensional        histogram, the two components u, v are not considered separately        from one another, but rather jointly.

In addition, the value ranges and the resolutions must also beestablished. The value range of the respective histogram is meaningfullyoriented to a global or to a local search area of the correspondenceformation, for example, at the minimal and the maximal locatablehorizontal movement in the optical flow.

The resolution in each case or the interval widths of the histogramintervals, which are also referred to as histogram bins, may, forexample, be oriented to the desired accuracy or to the resolution of thegiven hypotheses or to the amount of available memory.

The interval widths should not be selected to be too small, however;otherwise, there is the risk that in each case too little data fall intoa histogram bin or histogram interval and, as a result, thedetermination of majorities is hampered or becomes more complicated dueto lack of clarity.

In order to avoid or to reduce undesirable quantization effects, whichmay arise as a result of the given interval widths, it may beadvantageous not only to enter the value to be entered in thehistogram—for example, an increment +1 or decrement −1 or an additivesigned weight, into a bin, but to distribute it weighted to the twonearest bins.

Example: instead of the value u=9.4, to round off to 9 and to add +1 atthe appropriate point in the histogram, a weight +0.6 would be added for9 and a weight +0.4 would be added for 10 and thus, the total weight of1.0 would be distributed to two bins, specifically, with a weightingcorresponding to the intervals.

The respective histogram may be implemented as a sliding histogram orwith the aid of a sliding window, as this is also depicted in connectionwith FIG. 2. This saves computing time and data transfers from/to thememory, because the content of the histogram as a whole need not alwaysbe newly structured, but may be continuously updated.

The sliding histogram may be implemented by a sliding window, which has,for example, a rectangular shape or another shape, for example, theshape of a circle, of an oval, of an ellipsis, of a polygon, etc.

In addition to the histogram, it is advantageous to also provide amemory, in which the sum of all entries are continuously entered intothe histogram, i.e., the number of hypotheses situated under the windowor the sum of the weights situated under the window. This sum musttherefore be updated with each write access into the histogram.

Providing the sum memory offers the advantage that the number or thetotal weight may be retrieved at any time and need not be firstrecalculated. It may, for example, be useful for the purpose ofstandardization to determine, for example, how large the relativeproportion of hypotheses (under the updated window) is, that support aparticular movement hypothesis.

In two one-dimensional histograms for the vector components, it issufficient to provide the sum memory only once, since the sum of therespectively entered weights for both histograms in the preferredembodiment is identical.

Sliding Window (Exemplary Embodiment)

A, for example, rectangular window 20 according to FIG. 2 having width Band height H or a differently shaped window 20, for example, accordingto FIG. 3, is effectively pushed across image 10 and or the memory ofhypotheses or of its components. A snapshot is shown in each of FIGS. 2and 3. Window 20 as a sliding window is pushed, for example, in pixelcolumns from left to right across image 10 of correspondence hypothesesor across an image 10, each of which contains a component of thecorrespondence hypotheses, for example, the u-component. The movement ofwindow 20 is characterized, for example, by arrow 23.

Window 20 is located directly at the position marked by the black frame.Pixels 22 at the right margin of window 20 have therefore just migratedunder window 20 and pixels 24 have departed window 20 at the leftmargin. An update for precisely these pixels 22 and 24 is carried out.In the process, the hypotheses of the positions of pixels 22 are enteredinto the histogram by incrementing corresponding counters or by addingcorresponding weights for the individual histogram intervals or bins.The “opposite” update is carried out for pixels 24, specifically, in thesense that here the corresponding counters are decremented again or thecorresponding weights are subtracted with respect to the individualhistogram intervals or bins. In this case, it is ensured—when arespective update is completed—that precisely those hypotheses are takeninto consideration in the histogram that are directly situated underwindow 20, i.e., inside the frame of window 20. Hypotheses outsidewindow 20 on the other hand have no influence on the content of thehistogram.

In this case, one or multiple hypothesis(es) or also no hypotheses maybe present for each image position (depicted here as pixels). Forexample, 0 to 3 hypotheses per pixel may be stored. This means that whenupdating, these up to three hypotheses are entered into or removed fromthe histogram.

Pixel 1, 26 represents a reference position, which is situated, forexample, in the center of window 20. Pixel 1, 26 is the pixel to whichthe instantaneously created histogram relates. Statements, for example,for the purpose of verification, may be made for the hypotheses at ornear the reference position based on the updated histogram content, asis also explained below.

Arrow 23 indicates in which direction window 20 is further pushed whenthe histogram update step and the verification step are completed. Theorientation of window 20 and direction 23 of the pushing or sliding ofwindow 20 are preferably selected in such a way that window 20 has thesmallest possible “front end” or leading edge during the sliding orpushing of window 20, because then the number of pixels incorporated inthe update is minimal. In this example, 2-H pixels or image positionsare incorporated in the update. This number is smaller than 2-B.

A respective window 20 need not be situated completely in image 10, butmay also overlap at the margins. FIG. 4 shows how window 20 slides tothe left into image 10 and slides out to the right. Reference position1, 26 is situated here in the first image row.

This approach according to FIG. 4 may be advantageous, i.e., in thatwindow 20 is pushed step-wise into image 10, i.e., into the hypothesismatrix. In this case, it is never necessary to initially fill histogram63, 64. Instead, an empty histogram 63, 64—all bins are set to 0—may beassumed and only the updating process establishes any required fillingstate of respective histogram 63, 64, even at the image margins.

FIG. 4 shows by way of example how window 20 is pushed from left toright in the direction of arrow 23 and the overlap initially increasesin the process. The values for pixels 22 at the right margin of window20 are entered, whereas no pixels are yet present at the left margin ofwindow 20, i.e., nothing is yet removed. It is the reverse at the rightmargin of image 10. There, the values for pixels 24 at the left marginof window 20 are removed as long as there are no longer pixels presentat the right margin, i.e., nothing more is entered until histogram 63,64 is completely emptied, i.e., the initial state is reestablished.

Instead of allowing window 20 to pass completely out of image 10, it isalternatively also possible to carry out a reset, in which all memorylocations are set to 0 once the histogram content is no longer requiredfor a verification step.

Once a row is fully processed, the same process may be carried out inthe next row, for example. The process is also parallelizable.

FIG. 5 shows a specific variant for the processing of image 10 with achange of direction of sliding window 20, in each case to the right andto the left at the margin of image 10.

The direction of the sliding or shifting may thus also be changed, forexample, in each case if reference pixel 1, 26 has reached a margin ofimage 10. For example, window 20, starting from left to right, would beshifted to the right until reference pixel 1, 26 touches the margin, butis still situated in image 10. Then, window 20 would be shifted, forexample, downward by one pixel, i.e., the update process would becarried out by way of exception for the wide side. Window 20 wouldsubsequently be allowed to pass in the direction opposite the previousdirection, i.e., from right to left, until reference pixel 1, 26 reachesthe left margin of image 10, to shift again downward by one pixel andchange direction again, etc.

In the interim, the histogram would generally not reach the initialstate. This presents no problem, however, but even an advantage, sincefewer data need be entered or removed.

Histogram Data (Exemplary Embodiment)

FIG. 6 shows a typical snapshot of histograms 63 and 64 for thecomponents u and v, which are implemented here with a resolution(interval width) of 1 pixel. It is apparent that peaks 65 are formed,which represent the dominant movement within window 20. Peak 65 isdistributed usually over one or a few bins, depending on the resolutionof the histogram.

FIG. 1 depicts specifically a snapshot of the content of the twoone-dimensional histograms 63 and 64 for the horizontal (histogram 63)and vertical movement component (histogram 64) u and v.

Two or multiple dominant movements also exist in part within window 20when, for example, in the application with window 20, half correspondsto a pedestrian in the foreground of an image and the other halfcorresponds to the background of the image, overlap there, and thehalves exhibit different movements in the image. Then, correspondinglymultiple peaks 65, which represent these multiple movements, aregenerally found in the histogram.

Outside the dominant peak or peaks 65, the histogram bins are filledwith zero or with small values. These small values are usually caused byoutliers.

The representation with histograms 63, 64 is thus well suited forseparating the dominant and, therefore, likely correct movement from theoutliers, which appear rather statistically dispersed.

Weighted Histograms

Events are counted in a classic histogram 63, 64. As previouslymentioned, it may be advantageous, however, to consider hypotheses [u,v] as not each having identical weight 1, but having differentindividual weights. Thus, it is possible, for example, to take intoconsideration a measure for the confidence in the respective hypothesis.

An algorithm for each hypothesis not further considered here could, forexample, also specify an individual measure of quality, measure ofconfidence, weight or the like.

Such a measure may be taken into consideration with the entry intohistogram 63, 64 by replacing the histogram counter with a total weight.For this purpose, a corresponding value range must be provided. In thiscase, it is advantageous to continue to operate with integer values orfixed decimals in the sliding histogram, not with floating pointnumbers. It should preferably be ensured that deviations do not occur asa result of non-cancelling rounding-off errors during the ongoingaddition and subtraction. This risk would exist in the case of floatingpoint numbers. A potentially necessary rounding-off should be performedin advance, so that in the case of the sliding histogram 63, 64, it isensured that the entered weight is later removed again without aremainder being left occurring as a result of rounding-off.

In the case of weighted histogram 63, 64, the sum counter, whichindicates the instantaneously entered hypotheses, becomes a weight sumcounter, which indicates the sum of the weights of the instantaneouslyentered hypotheses.

Verification Step

Once the update has been carried out, as described above, the content ofthe histograms 63, 64, represents the movement hypotheses located undergiven window 20, the verification step may be carried out for thehypotheses at the reference position, and, if necessary, in the smallsurroundings around the window.

In the verification step, it is checked whether sufficient support fromthe vicinity is present for each hypothesis considered, the vicinitybeing established by the size and position of window 20 in relation toreference position 1, 26.

These hypotheses from this vicinity, to give a vivid example, have thuscast their votes, as in an election, and in each case have voted for aparticular movement, specifically, in components in two one-dimensionalhistograms 63, 64 or in vectors in a two-dimensional compositehistogram.

For the hypothesis to be verified, it is then checked whether it issupported by a majority or whether there is at least a sufficient amountof support.

In two-dimensional histograms 63, 64, this check is carried outindividually per component and the result is combined, for example, bydrawing upon the poorer of the two results or by linking the two resultsin another way to one another, for example by addition.

The step of checking in this case may appear as described below, thecheck being related to a hypothesis vector, which is assigned, forexample to marked position 1, 26 in FIG. 2 or FIG. 3, and thus includes,in particular, the following steps:

-   breakdown of the hypothesis vector into its components (omitted in    the composite histogram)-   reading access in the respective histogram 63, 64 at the position 1,    16, which corresponds to the movement (component) of the hypothesis.    If, for example, the u-component of the hypothesis is −1.7 pixels,    the histogram would be read out having interval widths of 1 pixel at    the point which stands for u=−2, optionally also at the point for    u=−1, because the value −1.7 is between u=−2 and u=−1.-   Histogram 63, 64 may optionally also be read out at other positions    located in the vicinity, for example at the positions −3 and 0. In a    composite histogram, a small, for example quadratic, section in the    vicinity would be read out.-   The read out values, which correspond to counters or weights, are    then combined, for example, added or added in a weighted manner, in    order to ascertain an evaluation value. The weights may be selected    as a function of (rounded or unrounded) values of the hypothesis. In    the case of the aforementioned value −1.7, for example, the weights    0.7 and 0.3 could be applied for the histogram entries read out    where u=−2 and u=−1. The histogram entry where u=−2 would therefore    be more heavily weighted, because −1.7 is closer to −2 than to −1.    All weightings may preferably also be performed using integer    values—for example, fixed point presentation.-   This evaluation value represents a first evaluation for the    hypothesis. The higher this evaluation is, the clearer is the    confirmation from the vicinity for the present movement. The value    would be low in the case of an outlier.    -   In two one-dimensional histograms, two evaluation values would        occur, which may be combined, for example, by considering the        minimum or the sum.-   This optionally combined evaluation value may be compared with a    threshold, in order to decide whether, based on sufficient support    from the histogram-based evaluation, the hypothesis is trustworthy    or not.    -   Alternatively or in addition, the evaluation value—if necessary        in quantized or coded form—may be appended to the hypothesis as        additional information and passed on, for example, in order to        be evaluated at a later point or to be provided at an interface.-   Instead of using an absolute threshold, it is alternatively or    additionally possible to also consider a relative threshold. For    example, the sum counter or the weight sum counter may be used in    order to determine how large the proportion of hypotheses from the    vicinity is which support the hypothesis just considered.-   In the histogram-based vote, it may be meaningful or desirable, to    discount the individual vote in each case, so that a hypothesis is    not able to vote for itself. This is easily possible by subtracting    in each case the individual weight when reading out the    corresponding respective histogram bin and the sum counter or the    weight sum counter.

Alternatively or in addition, the peak position may also be used forverifying a hypothesis vector:

-   -   A dominant movement may be ascertained at any point in time from        histogram 63, 64 or from histograms 63, 64, for example, by        establishing which movement corresponds to the dominant peak.        Which bin the dominant peak is to be assigned may, for example        be established as follows:    -   by the bin with the highest weight,    -   by the bin which, together with one adjacent bin, has the lower        or equal weight in each case, in sum bears the highest weight of        all such bin pairs,    -   by the bin, for which the highest weight is ascertained given a        weighted average over a predefined proximity of bins        (implementable, for example, by folding the histogram, for        example, using symmetrical coefficients [c2, c1, c0, c1, c2]        with c0≥c1≥c2≥0.

The two last mentioned points represent smoothing measures, which inpractice result in a more stable peak selection and are therefore to bepreferred.

-   The dominant peak position may be ascertained in various ways:    -   Full search across respective histogram 63, 64: this method        always finds the dominant peak, but requires greater effort,        since the entire histogram is read out.    -   Incremental search: after each change of the histogram content        or after a number of changes or after a certain period of time,        it is checked whether the dominant peak has “further migrated,”        for example, by one bin position to the left or to the right or        by a few bin positions. Only a small effort is required for this        check.    -   Check of the dominant peak position after the histogram update:        For each bin that is changed, it is checked whether in the        interim the dominant peak position is situated there (or        optionally in close proximity).    -   Comparison of the weights assigned the potentially dominant peak        with the sum weight. As long as the ratio is greater than ½, the        dominance is confirmed. This test is usually successful in        examples as shown in FIG. 5.    -   With a combination of the last mentioned methods, the full        search is only seldom necessary or no longer necessary at all.-   In addition to the “first” dominant peak position, it is also    possible to ascertain a second or second-best peak position, if    necessary, a third peak position, etc. This is advantageous and of    practical relevance, in particular, if multiple dominant movements    are situated within window 20.-   When checking a hypothesis, it is taken into consideration whether    or how close it is situated to the dominant peak position or to one    of the dominant peak positions.    -   For example, it could be required that both components of the        hypothesis are no further than 3 bins away from a peak in        respective histogram 63, 64 in order to accept the hypothesis.    -   Alternatively or in addition, a measure of quality could be        appended to the hypothesis, which includes a statement about how        well the hypothesis is supported by the dominant movement (or by        one of the dominant movements).

The results of the majority-based verification of correspondences have atypical “appearance”, for example, when results are visualized in color,as has been mentioned at the outset.

This is particularly true in the boundary areas when the method isapplied to difficult data, for example, in the case of weaksignal-to-noise ratio or movement unsharpness or in the case ofconcealment. The method is easy to recognize based on the type ofdegradation and on several characteristic properties, for example,

-   by the formation of clusters of results due to the mutual    confirmation of hypotheses,-   by sharp jumps at object edges (no smearing) due to a change from    one dominant peak 65 to another dominant peak 65 and/or-   by the positive efficiency when comparing input and output in the    sense that almost no outlier hypotheses remain.

In addition, the access patterns to the memory and buffer memory arevery characteristic when window 20 slides across image 10 of hypotheses,as is shown in connection with FIG. 2.

The present invention is suitable, for example, for implementing onCPUs, FPGAs and as an ASIC or an ASIC IP and may be implemented on allplatforms.

The method of optical flow, also called optical flow method (OF) is amethod for analyzing movement with or in conjunction with computervision methods.

A camera sensor is able to generate images that contain projected 2Dpoints of the 3D world. In OF, images are analyzed which have beenrecorded at different exposure times, and so-called correspondences areconstructed for those points whose coordinates may be localized in bothimages. A correspondence vector connects the coordinates of one pixel inthe first image with the coordinates of the same 2D point in the secondimage. The correspondence vector in optical flow is also called a flowvector.

The method described above describes the approach for identifying andconfirming or rejecting outliers of a hypothesis image 10.

The procedure described below explains as an additional aspect thepossibility of acceleration in the above described basic method, inorder to achieve a real-time processing during the verification processor in order to reduce the computing time for the verification process.In this procedure, the above described verification process is modified,which may lead to different results. The difference in the results inthis case is so minimal that when practically applied it results in norelevant disadvantages.

The methods described here may be used, for example, in theimplementation of computer vision ASICs.

In a processing throughput of, for example, 60 full HD images persecond, a driver assistance camera system must achieve a minimal powerconsumption of at most a few watts, in order to enable an automaticintervention system, for example, an emergency braking assistancefunction. In such camera systems, the memory bandwidth is not sufficientto carry out a histogram-based verification regarding everycorrespondence hypothesis. The size of the image area in which histogram63, 64 must be calculated determines the computing time and the time forthe processing of the algorithm. For large areas or regions, therequired time for updating the histogram and for analyzing the contentcould impair a system in processing images B1, B2 in real time in anembedded system. In the event the region is small, the peak ofhistograms 63, 64 is potentially not pronounced strongly enough in orderto reliably enable the distinction between outliers and correcthypotheses.

The present invention accelerates the method for rejecting or verifyingoutliers and makes it possible to achieve real-time applications such asan emergency braking assistance function in an embedded system.

The following described aspects reduce the processing time of ahistogram-based procedure for verifying the correspondence vectorhypotheses using the content of histogram 63, 64 instantaneously presentin a memory, multiple hypotheses being checked in parallel.

In the following description, it is assumed that the first step fordetermining the correspondences has already been carried out, so that animage 10 including correspondence hypotheses that must be checked isavailable in a memory unit. It is assumed that this image 10 containsvalid elements and outliers. Valid elements, also referred to asinliers, typically represent the true movement, whereas outliers arefalse correspondences. Only the outliers must be rejected and theothers, i.e., the former, must be preserved. This is typically the lastprocessing step so that its output is correspondence vector image 100,correspondence vectors being able to be optical flow vectors, forexample. One property of the outliers is that they are randomlydistributed, whereas the inliers are spatially supported by adjacentpixels. For this reason, a verification based on histograms 63, 64 is agood choice, since the inliers belong to local or global peaks 65 of ahistogram 63, 64.

It is assumed that a memory is used in order to store the values of thehistogram bins for a given area. There are multiple possibilities forstoring the values of a sliding window histogram in an electronicdevice. One possibility is to use the build memory within an IP havingbit memory cells, for example, flip-flops. These have the potential toaccess all bins of the histogram in only one clock cycle, but theircosts are too high to be taken into consideration in a product.

A further approach is to use the in-chip or off-chip memory havinguser-defined input and output ports. These memories have a limitednumber of ports for reading the values and for storing. When designingan electronic system including the corresponding required memory, acompromise must be found between the size of the required semiconductorarea (greater area has a strong influence on costs) and performance. Thememories may be selected from a single-port memory (only one readoperation or write operation per clock cycle) or from dual- to quad-portmemories (including two, three or four read operations and only onewrite operation in the same clock cycle). Since the quad-port memoriesare not available in all electronic devices and their surfaceconsumption is greater, the dual-port memories are the most frequent forapplications, in which a higher through-put is required.

In the above described verification method, an approach including asliding window 20 is used for rejecting outliers of the OF. It reducesthe data transmissions from the memory that contains the correspondencehypothesis image 10 into the memory that contains histogram 63, 64,since histogram 63, 64 need not always be recreated. Read operations andwrite operations in the histogram memory are reduced to double thenumber of rows in the area in which histogram 63, 64 is calculated. Arectangular sliding window having a height of 9 rows is shown as anexample in FIG. 7. To simplify, it is assumed for the followingobservation that exactly one hypothesis is present per pixel—in realitymultiple hypotheses or no hypothesis at all per pixel may also bepresent—and that each hypothesis contributes with weight 1. The updatingof a histogram then lasts 18 clock cycles due to the height of 9 rows,since 18 addresses must be accessed. The 9 bins of the new column thatenter into histogram 63, 64, must be incremented, and the 9 bins of theold columns that depart the histogram 63, 64 must be decremented.

Once histogram 63, 64 is updated, the next step is to identify whetherthe instantaneous correspondence hypothesis belongs to a peak 65 in thehistogram. This takes place by accessing some of the adjacent bins orhistogram intervals, followed by the comparison between the values ofthese bins. Since the updating and the evaluation of histogram 63, 64must take place for the entire correspondence hypothesis input image 10,the real-time processing of the above described procedure for largeimages is not possible due to the limited memory bandwidth in anembedded device.

The present invention also provides three methods for accelerating theverification of correspondence hypotheses.

The first acceleration method is to save clock cycles during theevaluation of histogram 63, 64. In the process, the memory storinghistogram 63, 64 is organized in such a way that one clock cycle issufficient for obtaining the up-to-date average pixel and up to four ofits adjacent bins or histogram intervals. This is shown in FIG. 8. Theread address must be calculated by adding or subtracting a value that isstored in the middle or central pixel. For example, by grouping eighthistograms into one address, it is possible to read up to eight adjacentbins in only one cycle. Given the fact that the middle pixel ofhistogram 63, 64 is assigned to bin number 11, for example, the readaddresses must be set as address number 0 for the first port and theaddress number 1 at the second port. Both ports together contain a totalof 16 bins, from bin number 0 to bin number 15. The first read portsupplies the fourth bin (bin number 7) on the left side of theup-to-date center (bin number 11). The second port contains the otherthree bins to the left (bins 8 through 10) and the other four bins onthe right side (bin 12 through 15).

The second acceleration method reduces the total time of the processingof the entire correspondence hypothesis image by using the content ofhistogram 63, 64 for verifying N correspondence hypotheses in parallel,instead of verifying only one correspondence hypothesis as in theoriginal proposal. A slight asymmetry occurs then because not allhypotheses may be situated in the middle of window 20 (which was 9 rowshigh in FIG. 7, for example), for which histogram 63, 64 is calculated.In order to partially compensate for this asymmetry, it is possible toincrease the number of rows, preferably by N−1 rows. Although increasingthe number of rows increases the time for updating and evaluatinghistogram 63, 64, the total time for verifying a complete correspondencehypothesis image 10 is reduced, since with a histogram filling state, itis possible to verify N hypothesis at the same time.

In this example, an increase in the effort by the factor(8+(N−1))/9=(8+n)/9 due to the increase in number of rows is offset by areduction by a factor N corresponding to the number of histogramupdates. On the whole, therefore, a factor (8+N) is to be considered inthis example by which the processing time of the algorithm is reduced.

On the other hand, the asymmetry also results in slightly changedresults as compared to the above described non-accelerated approachaccording to the present invention.

So that the results are not changed too significantly due to theasymmetry, a value of N=4 has proven to be a good compromise withrespect to the quality of the results and processing time. Where N=4 asshown in FIG. 9, sliding window 20 includes twelve rows and thehistogram update and histogram evaluation may be carried out in 24+4clock cycles, which results in a throughput of 28 clock cycles for theverification of 4 input correspondence hypotheses. The resultantthroughput of 7 clock cycles per input correspondence hypothesis enablesa real-time processing of the above described method or algorithm alsofor full HD images and at low clock frequencies, which are required fora low power consumption in an embedded device.

The third acceleration method is based on a reduction of the timerequired for updating histogram 63, 64, which may be advantageouslyapplied, in particular, when hypothesis image 10 is sparsely populated.Then, many of the incoming and outgoing hypothesis memories contain noinformation for updating histogram 63, 64. This property may be utilizedin order to reduce the number of rows for updating histogram 63, 64 to afixed number, which is smaller than the total number of rows of slidingwindow 20. For example, the number of correspondence hypotheses to beconsidered could be limited from twelve to six, taking intoconsideration sliding window 20, in which twelve rows are used to updatehistogram 63, 64, in order to accordingly reduce the number of clockcycles required also by half. In the unlikely case that more than sixhypotheses are available for contributing to the update, the remaindermust be skipped. In this case, it is necessary to define a prioritysequence.

One sequence for searching for the presence and for reading hypothesesfor the histogram update is depicted in FIG. 10. The six pixels numberedwith 1 have the highest priority and are always read. If some of thesepixels include no correspondence hypothesis, the next pixels are then tobe read, which are numbered with 2, 3, etc., specifically, until maximalsix correspondence hypotheses are read. It has been empirically proventhat the updating of histogram 63, 64 in this way achieves a similarquality of result as the updating of the histogram without skipping theremaining hypotheses.

The time required for verifying a correspondence hypothesis may bereduced from thirty clock cycles to nearly four clock cycles taking intoconsideration the three acceleration methods and a sliding window 20having a height of 12 rows, as a result of which a real-timeimplementation of the method or the algorithm in an embedded device isachieved, for example, at 60 full HD images per second.

1-15. (canceled)
 16. A method for evaluating images and for evaluatingcorrespondence hypotheses of images, the method comprising the followingsteps: providing correspondence hypotheses between first and secondimages each given as a corresponding image matrix, in a correspondinghypothesis matrix; evaluating the hypothesis matrix and conditionallyverifying the correspondence hypotheses; and providing verified imagecorrespondence hypotheses as image correspondences in a correspondencematrix as a result of the evaluation, the evaluation of the hypothesismatrix being performed in that for each element as a reference elementof the hypothesis matrix in surroundings of the reference element for atleast one component of the correspondence hypotheses, a histogram withrespect to values of the component is formed and evaluated.
 17. Themethod as recited in claim 16, wherein the formation of the histogramfor the respective element of the hypothesis matrix takes place byclassifying and totaling or totaling in a weighted manner the respectivecomponents of the correspondence hypotheses in a given histogramclassification for all elements of the hypothesis matrix within aninterior of the surroundings of the respective element.
 18. The methodas recited in claim 17, wherein when forming the respective histogram,the surroundings for the respective element of the hypothesis matrix areprovided in each case with a window overlaying wholly or partially thehypothesis matrix in the manner of a rectangle, of a polygon, of anoval, of an ellipsis or of a circle.
 19. The method as recited in claim17, wherein when evaluating and forming the histograms, all elements ofthe hypothesis matrix are captured by the respective surroundings or bya respective window, which are: (i) identical or similar for allelements of the hypothesis matrix with respect to form and/or extent,and/or (ii) configured as sliding surroundings or as a sliding window.20. The method as recited in claim 16, wherein when evaluating thehistogram for the respective element of the hypothesis matrix on thebasis of the histogram, an evaluation value is generated: by reading outone or multiple values of the histogram in at least one histograminterval for the correspondence hypothesis or of least one componentthereof assigned to the given element and by totaling them or totalingthem in a weighted manner to form the evaluation value, and by alsotaking into consideration one or multiple values from one or frommultiple histogram interval adjacent to the assigned histogram intervalor from the surroundings of adjacent histogram intervals.
 21. The methodas recited in claim 20, wherein when verifying a verification of thecorrespondence hypothesis for the respective element, the hypothesismatrix or a component thereof is identified, if the evaluation value atleast reaches a local or global predefined threshold value.
 22. Themethod as recited in claim 20, wherein when verifying the correspondencehypothesis to the respective element of the hypothesis matrix or of acomponent thereof, an assigned and verified correspondence or acomponent thereof is established in each case by an evaluation value andis entered as a corresponding element into the correspondence matrix.23. The method as recited in claim 16, wherein a respectivelyinstantaneously generated histogram, which is stored in a histogrammemory is used for the parallel and/or serial verification of multiplecorrespondence hypotheses for accelerating and/or for reducing aprocessing effort of the evaluation of the hypothesis matrix and/or ofthe conditional verification of the correspondence hypotheses.
 24. Themethod as recited in claim 23, wherein the histogram memory is organizedin such a way that an underlying clock cycle is sufficient in order toread out the value for a respectively up-to-date element of thehypothesis matrix from the histogram, as well as values for one or formultiple additional intervals of the histogram, (i) an underlying readaddress being calculated by adding or subtracting a value of 1 withrespect to the value for the up-to-date element stored in the hypothesismatrix and/or (ii) by clustering intervals of the histogram with respectto a dual port of the underlying histogram memory.
 25. The method asrecited in claim 16, wherein to accelerate the evaluation of thehypothesis matrix and/or of the conditional verification of thecorrespondence hypothesis: the conditional verification is carried outin parallel and/or simultaneously for a plurality of correspondencehypotheses for a corresponding plurality of elements of the hypothesismatrix, and/or not all elements of the hypothesis matrix within aninterior of the surroundings of the respective element and, of theunderlying sliding window, are taken into consideration when updatingthe histogram, but at most a predefined number, a priority sequence as afunction of the position in the window establishing in which sequencethe elements of the hypothesis matrix within the interior of thesurroundings of the respectively given element are primarily taken intoconsideration.
 26. A driver assistance system for a vehicle, the driverassistance system configured to evaluate images and evaluatecorrespondence hypotheses of images, the driver assistance systemconfigured to: provide correspondence hypotheses between first andsecond images each given as a corresponding image matrix, in acorresponding hypothesis matrix; evaluate the hypothesis matrix andconditionally verifying the correspondence hypotheses; and provideverified image correspondence hypotheses as image correspondences in acorrespondence matrix as a result of the evaluation, the evaluation ofthe hypothesis matrix being performed in that for each element as areference element of the hypothesis matrix in surroundings of thereference element for at least one component of the correspondencehypotheses, a histogram with respect to values of the component isformed and evaluated; wherein a result of the evaluation is used tocontrol operation of the vehicle.
 27. A device for evaluating images forevaluating correspondence hypotheses of images, the device configuredto: provide correspondence hypotheses between first and second imageseach given as a corresponding image matrix, in a correspondinghypothesis matrix; evaluate the hypothesis matrix and conditionallyverifying the correspondence hypotheses; and provide verified imagecorrespondence hypotheses as image correspondences in a correspondencematrix as a result of the evaluation, the evaluation of the hypothesismatrix being performed in that for each element as a reference elementof the hypothesis matrix in surroundings of the reference element for atleast one component of the correspondence hypotheses, a histogram withrespect to values of the component is formed and evaluated; wherein thedevice is an ASIC, or a freely programmable digital signal processingdevice, or a combination of the ASIC and the freely programmable digitalsignal processing device.
 28. An operating device which is configured tobe controlled during operation using a device for evaluating images forevaluating correspondence hypotheses of images, the device configuredto: provide correspondence hypotheses between first and second imageseach given as a corresponding image matrix, in a correspondinghypothesis matrix; evaluate the hypothesis matrix and conditionallyverifying the correspondence hypotheses; and provide verified imagecorrespondence hypotheses as image correspondences in a correspondencematrix as a result of the evaluation, the evaluation of the hypothesismatrix being performed in that for each element as a reference elementof the hypothesis matrix in surroundings of the reference element for atleast one component of the correspondence hypotheses, a histogram withrespect to values of the component is formed and evaluated; wherein thedevice is an ASIC, or a freely programmable digital signal processingdevice, or a combination of the ASIC and the freely programmable digitalsignal processing device.
 29. The operating device as recited in claim28, wherein the operating device is a vehicle.
 30. A non-transitorymachine-readable memory medium on which is stored a computer program forevaluating images and for evaluating correspondence hypotheses ofimages, the computer program, when executed by a computer or a digitalsignal processing unit, causing the computer or the digital signalprocessing unit to perform the following steps: providing correspondencehypotheses between first and second images each given as a correspondingimage matrix, in a corresponding hypothesis matrix; evaluating thehypothesis matrix and conditionally verifying the correspondencehypotheses; and providing verified image correspondence hypotheses asimage correspondences in a correspondence matrix as a result of theevaluation, the evaluation of the hypothesis matrix being performed inthat for each element as a reference element of the hypothesis matrix insurroundings of the reference element for at least one component of thecorrespondence hypotheses, a histogram with respect to values of thecomponent is formed and evaluated.